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Introducti on - * * ' , ^ 

Byety person Involved In any field of evaluation' evolves* Each person unQ^*-' 
goes changes ^In the process of his career* The process of change^beglns vhen one 
decides ^that evaluation is going" to be his or her.c^eer* For most this means ^^"^ — 
entering an academic institution* During this tiiiiey' each ev^luator is bor^* His" * 
heredity and* enfv^xonment depend primarily upon the institution (s) he attends* The 
evaluator Is taught and nurtured until tie can walk and talk* However 9. the' set of 
experiences he is exposed' tb are limited and controlle49 and he enters the next era 
of change 9 the ^'world of pifact^cing reality'* without the opportunity for practice* 
In^this world, changes begin to take place rapidly, pnd many of theto can be dra8tic**s 
One can only -hope the^'/teachings of y<iuth" are not totally forgottto and set aside* 
Each^ evaluator is in a different stage of development; some will progress faster 
than others, and some %A\\ either iremaln infants* or die, as functioning evaluators* 
For ^ach evaluator, the vls4>pn or dream bjE a "better" future lies ahead, and 'to a 

* " • • ' r> ' .r ^< ^ - 

great extent the future .can b^ no better t;han dreams and visions that *have been held • 
ifor .lt in* the past* , * ^ ^ ^ ' 

The Past - ' ' > 

^ Upon arriving fresh jfroni the academic world on^ brings manx utppldn 

ideas with the hbpe of improving the. educational field*' ^o task se^s too large or 

too impossible; it should only' be a simple matter of applying o|te of the many theories 

^ • ^ - «' * ^ , ' " ^ - - , 

models or designs that have been learned,- and putting it to task* It is thought that. 

• ' * *• » •> 

the Job of^belng a formative* evaluator will be; exciting, a position t|iat will als^ow 
free flowing exchange, and/or modification for improvement •Tlie evaluator thinks 
he is a co'-captain or Ist officer of the ship, the 'instigator for change, the person 
providing information and recommendations for a "better w^y*" All of this with the 
support of the product staff* Indeed, jt:he evaluator sees himself or herself .as a 
valuable asset tb gbod product development*' 



Unfox:tunrdtelyj, the^ world of textbooks and claseroomst often. has ^little con* 
nect'lpn to the "nitty gritty" of the real worldo Application often* goes sou^y ^ 
and principles sometime fade and become comproniised' as the ^ame oi^ give and "take 
begins;/^ . ' - , . , . \ i ' 

Insights into the future of the" evaluator can emerge^'wheri pne returns to 

• ' - V 

that eager young person about to .venture into the "real wprld" of R & D evaluatipno ' ' 

'-^^ ' ' . ' • , ^ * ^ . ^' ^ • * ' - 

^'cept f or a feur^obscure'tnissing elements t his concept of evaluation has been . 

• ' » ' • • . ^ ' ' * - , , 

statfed very niqely by Scriven 01967) o That is, evaluation involves the collection 
af 'information concerning the- impact of an educational program, - Ahd^ wh^le t;here 
are many possible uses for suj^h informatidn^ it is, assumed, by the new^eval^uator 
j^hat *the ftmdamental"pur|>ose of all evaluation is to produce information whldh can 
be used in* educational decision-making. These decisilms may be concerned with the 
continuatipn» termination^ or modification of an^ exlstitig programt or with th^ de- 
velopment and, possible adoption 'of some new prograzp* . This definition appears to be 

sufficient until ^one dj^scovers^ that those* "few obscure missing elements// for tjie 

\ * ' *i " 

most party can be translated into "the product developelj*." NonDally, Before one is 

hired^y t:he Impression he ^d "of his 5ob was to 2 1) familiarize himself with the 

product being developed, *2) test it in the field, 3) cdllec't the data, 4) analyze 

the data, and . 5). finally prea^^il[t/wh&c_has been discovered to the product developer 

or employer for ^decision-mfiklng purposeSo , 

\ , , r ' ' ^ 

W^en thQ evaluator commences als or her new Job, the flrsft task is to become * 
familiar with the product or program to be ^evaluatedo' One assumes lie or she is the 
specialist in tills area (at least tliat yas the impression conveyed at the^time of . 

"feiring). After this, one.lbeglns the procfess of selecting one of the experimental 

. ^ . ^ ' 

designs to teat the product. The design chosen has all of the best characteristics 

♦ ' » • • 

needed* for a "good" evaluation. If needed, the design will include randoinlzation 

of sublects to treatment, an "N" large enough to test for ^any treatment interaction (s) , 

*a balanced design, consultants to' help with special problems, the same, covariant (a). , 



for pach Subject, ind .cost;, too, will be kept' under contrcU ' Tliis evaluklon will 
bt,;ione. by the books. The right wly. And the evaluAtot 'will remain independent . 
and totally unbiased. .Unfortunately^, the process- Isn-'t q.uite that simple. , ^ 
The evaluatdr is to be the unbiased judg^ ~ the essence of ImiJartlallty. 
However, this product hits been nurtured from its; inception by the developer. He 
Is not unlike a protective parent who knows that his child must go to' the hospital 
for II delicate and risky'operation, but is ut^ceftalTJ -of the possible outcome. Ilie. 
devel<Aer, too, knows that the product must be evaluated, but the outcome of that 
""evaluation on his "creative effort" is as dreaded to him as the possibility of the • 

*JL > ■ . - ' ^ . ' ? * ' 

deatn*^ the child on the operating table is to. the .par.ent. With these facts in 
front of^the evaluator, he#or she knows that a head-on-collision is no^ unlikely; 
the'^^valuat'of "prepares for the collision .with all the dl-plomacy and tact . that is 
needed by a United Nations ambassador. The theories and-s'tudies previrfusly read 
no longer represent .reality, and the evaluator is suddenly f acfe-to-f ace' with a' 
problem that did not exist back in the ^^ssrootn. The present is here, and it 
must be dealt with noWo . n' . " • , 

The Present •/ - - , ^ ^ - 

Initially, the role of the evaluator is' ambiguous, for two mast ers,^ Research 
and.DevelbpRjent, must be served. .If the needa and demands of the two- are congruent 
the evaluator can proceed -effifciehtly and effectively. 'But frequently the require- 
ments of each are mutually exctusi^. -^e researcher is teek^ng evaluation, and 
the developer may primarily be beeklng confirmation. 

The basic confrontation is between the theoretical framework which results 
frpjn the evaluator' s academic: training and th«/ constraiJlts, demands, ai^ idiosyn-> 

* • 

cracies of reality. - It is the traditional battle between .theory and practice. 
. . ■ v.. ■ ' ■ ■ , ' 

^* ' The academician can- devise axioms and procedures for conducting research, in- 

• ■ > .. . ! i \J ■ • ■ . 

eluding -statements having to do with the random selection of^subjects, the admin- 
• . ■ . . • • ^ % • . . ' . - • 

istration of evaluation instrument's, ,.>nd the exact .p'cthod for implementing the- 



experimental treatment. But the practitioner must try to Implement the theoretical 
requirements of the academician with the parameters of a real-life settings The 
practitioner must deal with schools which refuse. to permit random selection of stu- 
dents, or whose definition of randomness is signif icantly different from what 
ts required for true Randomness. He or she must deal with school sj:rikes, ^arly 
closings, with the replacement of a teacher in mid-treatment, witH, mat'erlAis delayed 
or los^ in .'transit, with unexpected amounts of attrition, with students^ who were, tfon- 
tamlnated by previous participation in. a simil^^ evaluation f>ro'ject, and with the way 
costs can spiral and consequently cause original plants t.6.be modified in midstream. 
In fact, the possible ylolatlons to a tigHt research design are legion^ and it is 
experience which can^best help the evaluator to prepare for them* An important 
.caveat Is that one cannot predict everything tl\at will occur which may violate the 
design. HoyeveTif one can, attempt to control for as many as possible,- rqnemberlng 
that these violations are not ^^ones learned about- in the classroomo ^ 

Thferewls probably no evaluator who has not been, faced with these and even 
worse problem? in 1:he course of conducting an evaluation project. In fact, while 
these problems seem momentous to the new c^^uator, they may appeal, trivial tci 
exp^fenced evaluators. . Perhaps they are trivial in the face of higher order con- 
flicts betweei) the evaluator as evaluator and the evaluator as employee o^ the. - ^ 
.develop^:, fhe evaluator normally represents a certain threat to the ;3eveloperp 
^While the former may feel the need for conscientiously isolating variables, facts,- 
^and cpncluslpns; the latter may Intei^pret thlff.as being negatlveo' The unfortunate 
result of this perception is that the developer ;Day internalize this negativism 
and see it as an attack on the program or product, and therefore, by extension, as 
a personal attacks This then, is one of the basic confllctSo * . . ' . 

The friend vs. ft>e conflict that exists between the developer and the eval^- 
tor, when , it exists, may provlde the fuel for another taslc conflict which has a 
tremendous Impact on the findings of any .evalifatlonc.. This is the introduction of bit 



While the evaluator Is trained to be an objective observer, rcan^ 4evelopers 
have been erootlonallyrinvolved In thilr programs o Thus', a seemingly Innbcent dls- 
cuselon .of the merits of- a program can become filled with hidden agenda. The new 
evaluator wants to prove ^his worth by fljidlng out how successful tfte product Is, 
as veM as Identifying areas within the product or program which need to be cor- 
rented in the courle af its 4eveloptnertt ; but the developer wants to protect his 
"baby»» from attack. Far too often this results in a type of attack and defense / 
behavior. The dev.eloper may attack tfhe evaluator. on an instrument or the design 
to -be used in the evaluation. : Th^^ eviluator naturally defends his approach, and' , 
in' turn may attack the developer on some otKer point ^(e*g-. , the lack or unclarity 
.of the objectives ^r outcomes to be measured by the .evaluation) . ^This process of 
attack and defense causes undue bias . The objectivity of the evrfluator as w$ll ^ 
as the developer Is pat In jeopardy, knd the 'fevaluator, who may have been willing 
to modify his method, is now 'confronted with two. issues that Sxe hard to hwidle. 
One is that of his wo^Ch and integrity as a person.or .a professidijal, and the 
second ip. economic survival o . ^' , • - . * 

Obvliouslyi these are generalizations, but benea^ the objective facade of a 
developer insisting, "Of course I want to find out where and how 1 can improve thl 
program," there lurks aw artist," a creator, who doesn't want his creation to De 
stillborn. The goal of the evaluator, ^of course, is not to try to deliberately 

^ * , 

show the imperfections in the product, ^£e* se^ but to display the true nature of 
the product and help improve It in its developmental stages by documenting ^ts 

performance. " . ' . ^ 

If the developer* sees the evaluation as making a contribution tc the overall 
■ improvement of the pro^alh. If his long-range, objectives are seen as being con- 
gruent with those of fhe evaluator, if the results of the study are seen^^aQ being 
j^i^erim observations and not labels or Immutable conclusions;, then the basic ^coa-* 
flicts can. more readily be resolved. However, often, such is not the case.. 



Adnsittlng that the above intentions and purposee- are true is quite different, from 
the actual application and operation of these intents and purposes in the planning 
and decision-making, process* * • 

The authors have synthesize^ their experiences with those related to them by 
other evaluators and have listed a numbec of typical kinds of situations In which 
evaluators 9ft en f incJ^-Wee^lves. , ^- . * . » 

A 

1 » 

1) I No objectives:^ The specific objectives of the developers are not made 
clear so the, evialuator must "guess'' at wh^t spe^clf Ic elements are to be evaluated^, 
or the objectives and goals are under, continual revision and definition. Therefore, 
the emphasis of the evaluation Is based on the evaluator^s Interpretation of the 
problem* ^ . • 4 ' s 

■2) No communication: The developer may not be familiar with rese^arch ,teirm- * 
inology or procedures and may be reluctant to ask about them* ^Th^ evaluator may 
assume the developer Is aware, of these' things and may act accordingly, thus com- 
pounding the ^roblera.^ ^ ' * . . 

3) §uper-objectivity vs* nope:- The evaluator may alienate the developer by 
seralng to b,e distant, unconcerned, and cold-blooded instead of an' objective ob- 
sexrver* ' . ^ * • 

4) Poor timing: The evaluator is brought Into the picture too late f9r any 
changes* that may be necessary, and he Is unable to evaluate the product on its 
merits or demerits for improvement* Indeed, he may really be asked to dei^tjstrate. 
pjily one or the other* 

5) Bnphasis oti negativism: If the evaluator 's rol^ is seen as helping improve 
a program under development, the elements of the program which appear to be success- 
ful are often ignored and only those which show limited success are stressed* This 
emphasis makes the develpper feel threatened anc} defensive about the program. 



6) ^^fualvity of evaluators or dev^elQpers: Because evaludtors and deT^elop- 
• . , ' >^ ' • ■ 

^a'W not be able to conanunicate* successfully^ with each other, they tend to discuss 

'f . " ' " ■ 

their, vork with only other evaluators or developers. This reinforces the feelings of ^ 

-v • ■ , - ^- • 

"apartness" and reduces the opportunity for .lmpr6vinfecoinmanlcatlon-fdth each other. 
Evaluators and developers leirn from other evaluatorf and developers^, but they need . 
to share their knowledge with each o-hcr." 

.7) Protection of feelings and protection of job: Because the evaluator has 
often been\hired by or assigned to a, developer there ?nay exist the feelings that one 
should not be the bearer of unpleasant or even of non-positive tidings for fear of 
Offending: another professional or for' fear of l^elng fired for not producing suppor- 

tlve Information. • . ^ 

. When these are combined with a more exhaustive list prepared and discussed by 
Sanders and Gtiba X1973) the complexity of the job of the evaluator Is indeed made 
manifest. The number of conflicts that must.be facfed seems almost .endless, but 
they must, be confronted and mastered; After determining the natur6 of the conflict (s) 
' between the xoles of evaluators' and developers, oneanust plan bow to eliminate any" 
existing situations and how to prei^t such 'situttipna from beginning ^n the future. 

The Euture ^ ' . ^ ' ^ . - \ 

' The past expectations and the present demands- oi, the evaluator have probably , 
been somewhat, buJ not entirely^ exaggerated in this discussion. While it is true. 

• that every new evaluator as he leaves the world of academia may not have" all of 

■ ■ .■ > \ 

these expectations, it is likewise oerCaln that every evaluator leaves with some 

If n6t many of then. Similarly, all .the problems alluded to in the "present" «:e 

not faced, by each evaluator on each project, but that one or more of these "prob- 

leras" or "conflicts" or an adaptation of them, will at some time have (to be dealt 

if ' ' 

with .is almost a certalntyc ^ 

• Mow each person reacts' to the frustrations crjeated when he or she meets with 
the conflicts of expectancy or theory, and yiit\i the dally demands of' "practical 



reality** (i.e.f how the system Is really being operated) will. In the long run, 
xletermine the future t>£ the evi^uatqr .as a functioning member of the development > 
teamo If one feel^s that nothing c^n be done^ except ac^cept the "functioning 
reality** of product evaluation, then his or her contribution to '."real" evaluation 

Is, at best, limited and many times totally fruitless* On the contrar^ if 
one sets .the theory or ^expect at ion of "good" evaluation high, t;hen «ln spite 
.of any cohs^uences that may follow, "better" evaluation will be the inost 
probable outcome, ^ ^ . * ^ ' 

Achieving •'better" evaluation * to a large extent merely means the controlling 
of biases that ^e p^ lient or that can be Introduced into any evaluation which 
is to be conducted. Bias, much like Benhuda grass, is rooted in more than one 
place, ^ and it is hard to control. However, one. way to locate a large set of roots 
contributijig to evaluation bias is to look at the organizational model under which 
the evaluator must function (see also Scriven, 1974), 

Pbr the pizrpose of this paper^a Section Developer is the perion developing 
or supervising the creation or development of an educational product (e«g«, new 

curricula, strategy^ process, etc, ).« A Program Director is one who directs tw 

« 

or more Section Developers; a Division Director is in. charge of two or more Pro- 
grams; and the Cprporation consists of two or more divisions or programs, depending 
on the size of the Corporatfon, SolJ^dines -represent both supervision 'and budget- 
^ary responslbilltj^. Dotted lines represent inrormation and/or indirect supervision 
responalbilitie^, but« not budgetary control, (Refer to Models 1 thxo ugh 5), 

Scriven (1974) has stated two general principles that must be kept in mind 
if bias is* to, be minimized. The first is the, "Principle of Independent Feedbacl^," 
and the second is the 'Principle of Independaoce Maintenance,^' The first principle 
requires that ^*no unit in the chart [operat;ional hierarchy! should' rely entirely 
for evaluative feedback about a given sub-unit on the pl|»e down which it pours 
the money for that same siib-unlt" (Scriven, 1974), The second principle requires 

-9- 
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evaluation' of the evaluation because without it, Independence, whenever It exists, 
tends to be temporary in any bureaucratic structure* 

It should be obvious that as. the modeL number increases the likelihood of ^ 
controlling bias in any evaluation also increases© It is not until you reach 

# 

the second model that you even obtain the first principle, and Model 3 is the . 
first time for the second principle o It is not until Model 5 that both prin- 
ciples are met for the corporation, and then the first principle is violated fof ^ 
the funding agency. However, it should be easier for the funding agency to have 
greater "cohfiderice" in the results, findings, and decisions reached in Model 5 
tlian in any other model presented^ 

Since it is unlikely in any event that bias will be controlled or eliminatedl^ 
merely by the ^option or implementation of any of t\ia models described, it be- 
comes necessary to look for other means to achieve ''better" evaluation results 

f * * » 

and controlling biases* . ^ 

The following represent poWible suggestions for im^ementatibn to* achieve 

"betteu" evaluation r The first and most obvious, is the need to have better 

training programs (Worthen, 1^72 and ScriVen, 1974) o Not only must the evaluator 

receive better training in the what^ why, and how of evaluation, but this training 

must include more doing of evaluation o Each ^valuator must not only be taught* how 

I 

to waik and talk, but^he must be given a char^ce to practice the new found '"powers" 

/ ' ' . 

or "skillso/ In this way he or she is more likely to know what ^^is going to trip , 

him and how to move, circumvent, or eradicate the object that tripped him* New 

evaluators, as well as seasoned ones, "fail to produce satisfactory evaluations 

becaise they have not been equally well tutored in" the practical problems that 

they. are likely to encounter under real world conditions" (Sanders & Guha, 1973) • 

Indeed, many of these probl^n transcend ^most technical aspects of any evaluation 

to include the introduction of more general or pervasive v^riable'so Here is the 

place for academic institutions and institutions, involved in R 6e D to cooperateo 

• * * * 

-10- ' ■ ' / .• 



• . ' ' /■'■ ' ' ■ ' ' ' 

The R £c D lifstltutlons need to make ^ commitment » and to this extent mo must, 
the funding agencies, to assist 1^ the training of new evaluatorso The academic 
Institutions must In turn require 'practical experience as part of any degree 
program. This Idea Is not a new one; It has been practiced' for years In the , 
training of doctp^s, pharmacists, teachers^ and skilled c^k^aftsmen of' all tra<ieSo 

' ^ A second method of ^improving evaluation is the creation of evaluation guide'- 

lines. Worthen '(1972) and Scrlven (1974) are just two^ of many ^6 advocate this 

* * \ < ' ' ' ^ . . 

1 position. Gc6d guidelines would aid evaluatjlon and help control bias in. at least 

two ways: 1) thevevaluator would haye a base from which he can develop*^ com- • 

, prehenslve evaluation plan; 2) It would establish with the developer the require- 

^ ment thSt }:he evaluator must perforjn certain activities and that these activities 

are done to -achieve more usable results fi:om which to base d'eci-sions. 

Third, evaluation must be included from the beginning of the development 

project (Worthen,- 1972).. This would help alleviate many of the problems 'en coun- 

' • / ' . / - ^ ' ' u ' 

tered by* the evaluator. (e.g., janmeasmrable objectives or goals, arriving on the 

■ • \ ' 

qcene too late, etq.); , ' v 

Fourth, every evaluator should recognize that no matter how hard he or she 

tries to.be objective it is practically impossible to eliminate all the biases he 

^or she has or that are some way forced upon him or her. Consequently, every possible' 

^ ' t' ♦ 

' effort ahould be n/ade to supplement his own evaluation efforts With some type of 

outside evaluation. ^These might include one or more of the following as resources 

permit: ,Y) Go^l-Free Evaluation (GFE) as proposed by Scrlven (the articles 

^ referertced in the back of this paper provide a good explanation of GFE, its 

purposes and objectives, and dated 1972, 1973, & 1974); The Adversary Hearl^ 

^ ' (h gooH discussion of this* process is in the article by Owen, 1973); or con- 

tract ing of part of the evaluation outside the parent unito 

.A fifth way of controlling, bias (mainly the bias of how the evaluation data are 

used in decisipn-Tnaking for product modification) is to bring in another developer 

/ 

Q f ' / -11- < 
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in the field* .This developer would review the evaluation data and make ^Qcommen- 
dations in light of th?it data. This may be hard for some to accept, bu^ it is 
asking no more of the developer than the developer asks of the evaluator when 
he calls in a consultant to review the evaluator. If you wish, this might be 
teirmed Devfeloper-Free Evaluation. 

Last, and most important, the evaluate. ..id the developer should not look at 
each other as members of opposing teams who are trying'^ score the most points. 
They are really both members of the same team, holding different positions-, but 
whose end purpose is the smne. One should not be considered superior to the other 
but that each has an important and essential function to perform if a good educa- 
tional product is to- bp producedp 

Sinnmary 

Evaluation performs a -unique function in product development as well as in 
establishing product worth and effectiveness* The elimination of bias is par- 
ticularly hard in most organizations where product development is conducted, but 
its control is to be strived for at all levels and by^ each concerned party. The 
establishment of safeguards coupled with a conscientious effort to control biases 
will aid both in the better utilization of evaluation findings and the prevention 
of their misuse. Research, Evaluation, and Developmenc personnel should not be 
considered as different or opposing forces in product development, but as ^ team 
striving for creative, efficient, effective, anJ worthwhile educational products 
and prograirSo 
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Model 1 



Program Director 

r- L: — 



Section 1 Developer Section 2 Developer 

r- 1 , ^ 1 , 

Writers, etc. Evaluator Writers, etc* Evaluator 



, Model 2 



Program 1 


irector 




1 ■■" 1 


I 



Section 1 Developer Section 2 Developer Program Evaluator 

J U J J 1 V J 

ters, etc. (evaluator 1) writers, etc. (evaluator 2) evaluator 1 eValuator 2 



Model 3 ' • / 



This model could be the same Models 1 or 2, with the added 
dimension o£ Division or Corporate evaluation of the evaluator on 
a periodic basis. 
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Corp, or Divisi6n Evaluation 
I 
I 
I 

— -Evaluation Review Panel (A) 

(All evaluators in the Corp- 
oration are member q and serve 
on a rotating basis on small 
group review panels)' 



\(A) j(Review proposed design, 
instruments , etc. and make 
suggestions and comi^ents 
review reports before dis- 
tribution for adequacy and 
comnentF.) 
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